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Traditionally, researchers, policymakers, and practitioners have perceived a tension between 
rigor and accessibility in quantitative research and evaluation in postsecondary education. 
However, this study indicates that both producers and consumers of these studies value high- 
quality work and clear findings that can reach multiple audiences. The authors discuss the 
perceived importance of rigor and accessibility, barriers to accomplishing both in practice, 
and suggestions for achieving an appropriate balance between the two. 



Introduction 

“I’ve looked at academic research, and it’s just 
kind of amazing, the methods section and all 
their charts with the Rs and stats stuff, and 
that’s kind of irrelevant to people like me. You 
want to know that it’s a good study, well done, 
and that what they’re reporting is actually 
happening, that they can prove that it’s being 
shown, but it’s really a messaging thing. Even if 
you use a very rigorous method, you can also 
have very clear results’’ 

-State government staffer 

As budgets get tighter and postsecondary 
institutions face growing challenges in serving 
students and other stakeholders, administrators 
and policymakers confront difficult choices 
about what programs to preserve, expand, 
or cut. Evaluation and research are crucial in 
helping make these difficult decisions. Quasi- 
experimental designs, which we define as 
evaluation and research designs that resemble 
experiments but lack random assignment, are a 
popular way to evaluate postsecondary education 
initiatives. Comparison groups, for example, offer 
one way to estimate program effects through a 
quasi-experimental design. 



Quasi-experimental studies also are prevalent in 
postsecondary education research. But findings 
from these studies are not always “useful and 
used’’ in a chaotic policymaking and practice 
landscape that is saturated with information. 

In fact, the literature on the disconnect 
between policy and research discusses the 
“two communities’’ problem, which posits that 
research and policy communities have distinct 
languages, values, and goals. 
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It is easy to infer that producers of research and 
evaluation may prioritize rigor in educational 
research and evaluation while consumers may 
prefer “utilization-focused” studies that present 
easily understood findings tailored to their needs. 
However, a recent study we conducted on quasi- 
experimental method selection suggests that this 
perception is overly simplistic and that consumers 
and producers of research and evaluation share 
the belief that rigor and accessibility are crucial. 

Our Study 

In the summer of 2011, we undertook a small- 
scale study to examine the factors that influence 
the selection of quasi-experimental quantitative 
methods in postsecondary educational research and 
evaluation. In our time working at WISCAPE, we 
have served as both producers and consumers of 
research and evaluation using quasi-experimental, 
quantitative methods. We also have worked with 
a variety of policy actors — including campus 
administrators, faculty, and staff; legislators; 
government agency staff; and K-16 practitioners — 
who create and read this work regularly. In these 
experiences, we realized that individuals outside 
of the research community sometimes lack 
understanding of the method selection process and 
often find the work difficult to absorb. 

By undertaking this study, we hoped to illuminate 
the method selection processes employed by 
researchers and evaluators as well as explore how 
policymakers and practitioners view and use 
quasi-experimental research. By identifying places 
where the expectations and assumptions of these 
groups do not align, we can illuminate reasons for 
the policy/research disconnect and point towards 
strategies to improve the nexus between research, 
policy, and practice. 

We gathered data in the following three ways: 

1. A review of research published in various 
electronic databases and websites; 



2. An online survey of study participants, 
which focused on five common quasi- 
experimental methods in postsecondary 
education evaluation research 

3. Semi-structured interviews with 
participants informed by the research 
review and survey responses 

We chose our participants in this study to be 
representative of two groups: 1) researchers 
(scholars and other experts versed in quantitative 
research methods) and 2) practitioners 
(policymakers, administrators, and program 
staff). All participants have conducted work in 
or had some exposure to research and program 
evaluation in postsecondary education. 

How We Framed Rigor and 
Accessibility in Our Study 

As part of our study, we asked participants 
explicitly about rigor and accessibility in 
postsecondary research and evaluation. In the 
online survey, we asked participants to rate 
the perceived rigor, ease of understanding, and 
ease of execution of five commonly used quasi- 
experimental methods. 

In general, participants noted an inverse 
relationship between the perceived rigor of 
a method and the ease of understanding and 
executing it as well as an inverse relationship 
between perceived rigor and their familiarity with 
the method. For example, among the methods 
presented, participants perceived regression 
discontinuity and propensity score matching 
to be the most rigorous options, but they also 
ranked them as the methods with which they are 
least familiar and that are the most difficult to 
understand and execute. 

In contrast, participants believed that the options 
they perceived as less rigorous, like the use of all 
non-participants and univariate matching, were 
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easier to understand and execute. Our literature 
review also pointed toward this tradeoff: methods 
like propensity score matching and regression 
discontinuity appeared more often in venues stuck 
behind paywalls or unfamiliar to those outside 
of the academy (e.g., JSTOR) and infrequently in 
publicly available places that target policymakers 
and practitioners (e.g., the National Conference of 
State Legislatures website). 

Although our surveys and 
literature review supported 
the general perception that 
there is a tradeoff between 
rigor and accessibility, our 
interviews with participants 
painted a different picture. 

During these conversations, 
we asked whether it is 
better to prioritize rigor or 
accessibility in educational 
research and evaluation 
and about the tradeoffs, if 
any, between the two. This 
question prompted lively 
discussion and divergent 
opinions about the extent of the tradeoff and 
where researchers and evaluators should focus. 

The Importance of Rigor 
and Accessibility 

Our participants articulated several reasons 
why researchers and evaluators should pay close 
attention to the rigor of their methods. First, they 
believed rigorous designs instill confidence in 
researchers’ conclusions: the greater the rigor, 
the more likely the results capture the precise 
impact of a program. As one researcher noted, 
“Rigor is really important because you want to 
make sure that what you’re finding out really 
exists.’’ Second, they stated that rigorous designs 
are more likely to stand up to scrutiny from the 
scholarly community. Finally, participants argued 



that researchers always can make rigorous work 
more accessible in later iterations, but they cannot 
add rigor later in the process. As one practitioner 
stated, “I always would rather have the data be 
rigorous and comprehensive and precise because 
I can probably help interpret that or explain that 
to the end users in a layman’s way.’’ That sentiment 
was echoed by another researcher who argued 
that “you can always take rigorous results and 
make them accessible.’’ 



At the same time, our 
participants also called for 
researchers and evaluators to 
focus on creating accessible 
findings. They argued that 
accessible work can be more 
useful to those positioned 
to influence policy and 
practice. Our participants 
believed that policymakers 
and practitioners often do 
not understand rigorous 
social science methods, and 
this lack of understanding 
detracts from the face 
validity of the work. As one researcher asserted, 

“If your audience is someone who is not going to 
have the time or means to look up what you’re 
doing and fully understand it, then it’s more 
important to make it understandable to your 
audience, assuming you’re not sacrificing the 
entire integrity of the project.’’ Participants stated 
that prioritizing accessibility sends the message 
that engagement with communities outside of 
academe is an important goal and that research 
and evaluation can contribute to the public 
good. As one participant stated, “It’s not just the 
research but how it’s delivered, who it’s delivered 
to, and when it’s delivered.’’ 

Ultimately, though, most participants contended 
that evaluation research should strive to be 
rigorous and accessible no matter what methods 



...prioritizing 
accessibility sends 
the message that 
engagement with 
communities outside 
of academe is an 
important goal... 
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are chosen. As one researcher explained, 

“Rigor doesn’t have to come at the expense of 
accessibility, and vice versa. I think with the more 
recently popular designs, such as regression 
discontinuity and propensity score matching, it’s 
absolutely possible in my mind to make these very 
accessible information [sic], even if the audience 
doesn’t have to know how to do it, doesn’t have 
to understand all the statistical formula [s] behind 
those, conceptually they can be accessible.’’ 

To participants, accessibility 
comes from improving 
dissemination and 
presentation, not from making 
different methodological 
choices. As one participant 
stated, “Rigor is really 
important... but I am a huge 
fan of making that work 
accessible.’’ According to our 
participants, the two concepts 
must work in tandem because, 
as one researcher said, “I’m 
not sure what we’re making 
accessible if it isn’t rigorous.’’ 

In our interviews, we found 
rigor versus accessibility to be a false dichotomy; 
an ideal study combines sound methodology with 
clear descriptions of findings. 

What Stands in the Way? 

Our participants called for attention to both rigor 
and accessibility in evaluation and research, but 
still we are struck by the obstacles that may stand 
in the way of accomplishing this goal. For one, 
the incentive structures for faculty researchers 
tend to prioritize rigor over accessibility, however 
these terms are defined. The push for increased 
publications in top-tier, peer-reviewed journals, 
particularly for faculty members seeking tenure, 
may lead more researchers to choose methods 
that are considered highly rigorous but are not 
as accessible to policymakers and practitioners. 



As one participant suggested, “You can put 
propensity score matching in your title and get 
accepted [for publication].’’ Researchers may 
feel pressure to focus on rigor due to the need 
to create replicable studies and contribute to a 
generalizable body of knowledge. 

Conversely, evaluators may face pressures to 
create studies that conform to stakeholders’ needs, 
and stakeholders may prioritize simple answers or 
an accelerated research timeline over rigor. They 
also may confront budgetary 
or feasibility restrictions that 
make certain rigorous designs 
extremely difficult if not 
impossible to implement. 

Finally, the different 
skill sets and interests of 
researchers and evaluators 
also could make the balancing 
act between rigor and 
accessibility difficult. Both 
groups may use similar 
methods in their analysis but 
approach the communication 
and dissemination of findings 
in different ways. Academic researchers may 
have little interest in reaching policymakers 
and practitioners, instead seeing their primary 
audience as other academics. But academic 
researchers who wish to reach beyond academe 
may be accustomed to using jargon and lengthy 
explanations when writing about methods like 
propensity scoring and regression discontinuity 
for an academic audience, creating obstacles 
for policymakers and practitioners who lack 
technical background. As one participant stated, 
“The simple fact of the matter is I have yet to 
meet a policymaker who enjoys thinking in 
terms of multivariate distributions and t-values 
and t-statistics....They want to know yes/no, up/ 
down, viable/not viable, red/blue... that’s what 
they’re looking for.’’ 



. . we found rigor versus 
accessibility to be a 
false dichotomy; an 
ideal study combines 
sound methodology 
with clear descriptions 
of findings... 
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Conversely, evaluators may not prioritize or be able to 
publish their work for the larger research community 
due to confidentiality restrictions, client wishes, or 
the overall goals of the project. As such, their findings 
may not reach the scholarly community. 

How Do We Prioritize Rigor 
and Accessibility in Practice? 

For research and evaluation to have the greatest 
possible impact, we believe all producers of this 
work should prioritize rigor and accessibility. Our 
study suggests that many policy actors share this 
belief. But the persistence of this issue through 
decades of education research and evaluation 
shows that this will not happen without the use 
of clear strategies that help all groups treat both 
objectives with equal importance. We offer four 
suggestions to help make rigor and accessibility 
priorities moving forward. 

First, researchers and evaluators should consider 
using various structural techniques to help 
communicate rigorous methodologies and 
findings to those lacking statistical training so 
that the results can be utilized. Researchers can 
implement a variety of formats and elements 



in their papers — such as executive summaries, 
appendices, and other practices — that will 
highlight the most relevant findings at the front 
while leaving the technical components in place 
for peer review. When possible, researchers 
should reduce or define academic jargon, which 
will help make final products easier to understand. 
Academic researchers also can learn from the 
work of their evaluator peers who understand 
rigorous methods but may have more experience 
creating accessible products. 

Second, researchers and evaluators should 
consider publishing in a variety of venues to 
reach the widest audience possible and increase 
the likelihood that those inside and outside 
the academy will find and use their research. 
Faculty incentive structures encourage academic 
researchers to publish in prestigious peer- 
reviewed journals. Flowever, these journals often 
do not reach policymakers and practitioners 
who, according to our participants, prefer 
receiving information in more concise and 
straight-forward formats. As such, editorials, 
policy and research briefs, and trade publications 
are excellent venues for high-quality work aimed 
at these individuals. 



Recommendations at a Glance 



In working to prioritize both rigor and accessibility, researchers and evaluators should 
consider the following recommendations: 

1. Use various techniques (executive summaries, appendices, etc.) to help 
communicate rigorous methodologies and findings to those lacking statistical training 
so that the results can be utilized 

2. Publish in a variety of venues to reach the widest audience possible and increase the 
likelihood that those inside and outside the academy will find and use research 

3. Look to others to help translate findings and disseminate them more widely 

4. Engage with consumers of research and evaluation to better understand their 
conceptions of rigor and accessibility 
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Third, when it is not possible for researchers to 
create accessible products on their own, they 
should look to others to help translate findings 
and disseminate them more widely. For a variety 
of reasons, many researchers see their primary 
audience as the academic community, not the 
broader world of policy actors; as such, the 
interpretation and application of findings by 
practitioners and policymakers are not their chief 
concerns. In addition, some researchers have 
limited training in policy- or practice-focused 
writing. In these cases, various units inside 
postsecondary institutions, public policy think 
tanks, and other non-profit groups, for example, 
could provide assistance in crafting accessible 
summaries and connecting researchers to 
policymakers and practitioners. 



Finally, producers of evaluation and research need 
to actively engage with consumers of their work 
to better understand their conceptions of rigor 
and accessibility. Because these terms can mean 
different things to different groups, it is crucial 
that producers consider both the desired impact 
of their work and the needs of their intended 
audiences. Candid discussions about best 
practices and areas for improvement will tackle 
critical questions concerning methods, formats, 
and publication venues. These conversations 
will not only increase research quality and 
policymaking but also will lead to more 
meaningful relationships among researchers, 
policymakers, and practitioners that are built on 
trust and mutual understanding of the interests 
and needs of each group. 



I 
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' We would like to thank Mark Connolly and Annalee Good for their thoughtful review of the original draft. The 
authors bear sole responsibility for the content of this paper. 

^Thomas E. James & Paul D. Jorgensen, "Policy Knowledge, Policy Formulation and Change: Revisiting a Foundational 
Question," Policy Studies Journal 37, No. 1 (2009): 141-162. 

^ Robert Birnbaum, "Policy Scholars are from Venus; Policy Makers are from Mars," Review of Higher Education 23, No. 

2 (2000): 119-1 32; Erik C. Ness, "The Role of Information in the Policy Process: Implications for the Examination of 
Research Utilization in Fligher Education Policy," Higher Education: Handbook of Theory and Research 25 (2010): 1-49. 

''See Lant Pritchett, "It Pays to be Ignorant: A Simple Political Economy of Rigorous Program Evaluation," Policy Reform 
5, No. 4 (2002): 251-269; William R. Shadish, Thomas D. Cook, & Donald T. Campbell, Experimental and Quasi- 
Experimental Designs for Generalized Causal Inference. Boston: Floughton Mifflin, 2002; and Michael Quinn Patton, 
"Utilization-focused Evaluation," in Evaluation Models, edited by D.L. Stufflebeam, G.F. Madaus, &T. Kellaghan, 
425-438. Boston: Kluwer Academic Publishers, 2000. Pritchett (2002) argues that a lack of rigorous evaluations 
has contributed to general ignorance in policymaking while giving no attention to the presentation of findings as a 
potential explanation for this ignorance. Common evaluation texts like Shadish, Cook, & Campbell (2002) also focus 
much more heavily on issues of rigor and validity than on accessible presentation. Flowever, other authors, such as 
Patton (2000), make a compelling case for "utilization-focused evaluation" that is still methodologically sound but 
focuses on specific real-life uses for consumers. 

'’The authors presented results from this study at the Council on Public Policy and Higher Education Pre-Conference 
Forum at the Association for the Study of Higher Education's Annual Meeting in Charlotte, North Carolina on 
November 16, 2011. More information on the study, including a longer discussion of methodology and findings, can 
be found here: Elizabeth Vaade & Bo McCready, "Qpening the Black Box: The Process of Selecting Non-experimental 
Evaluation Methods and the Impact on Postsecondary Education Programs and Policy." Paper presented at the Council 
on Public Policy and Higher Education pre-conference forum at the annual meeting for the Association of the Study of 
Higher Education, Charlotte, North Carolina, November 16, 2011, http://go.wisc.edu/ljOdjh . 

®The following five methods were included: 1) using all non-participating individuals as a comparison group, 2) 
univariate matching, 3) multivariate matching, 4) propensity score matching, and 5) regression discontinuity. 

^We chose not to distinguish between researchers who focus primarily on academic or policy research and those who 
concentrate more on evaluation. Evaluators and academic researchers face distinct pressures and priorities in their work 
because they often serve different audiences. But despite these differences, evaluators and researchers use many of 
the same methods to complete their analyses and face similar considerations about how to create high-quality work. 

In addition, most of the researchers we spoke with have worked in both capacities— producing research for academic 
audiences and evaluations for clients— so they felt comfortable speaking broadly about method selection. For our 
study, the benefits of emphasizing the similarities in the challenges faced by academic researchers and evaluators 
outweighed the limitations of such an approach. However, future studies should distinguish between these groups 
carefully, depending on the research question. 

® In our study, we did not provide participants with definitions of rigor and accessibility; instead, we allowed our 
participants' personal definitions of these terms guide their responses. During our interviews, various interpretations 
emerged, and we pulled out some major themes that helped define the terms as we moved forward with our analysis. 
In general, we, along with our participants, saw rigor as encompassing the technical quality of research and evaluation 
relative to the question being asked, while accessibility is a multifaceted concept that includes both literal access 
to knowledge (including whether individuals can easily acquire a copy of the work) and the way that knowledge is 
presented (including whether individuals reading the work can easily comprehend what is written and presented). 
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